An empirical evaluation of imbalanced data strategies from a practitioner's point of view

This research tested the following well known strategies to deal with binary imbalanced data on 82 different real life data sets (sampled to imbalance rates of 5%, 3%, 1%, and 0.1%): class weight, SMOTE, Underbagging, and a baseline (just the base classifier). As base classifiers we used SVM with RBF kernel, random forests, and gradient boosting machines and we measured the quality of the resulting classifier using 6 different metrics (Area under the curve, Accuracy, F-measure, G-mean, Matthew's correlation coefficient and Balanced accuracy). The best strategy strongly depends on the metric used to measure the quality of the classifier. For AUC and accuracy class weight and the baseline perform better; for F-measure and MCC, SMOTE performs better; and for G-mean and balanced accuracy, underbagging

Empirical evaluation of resampling procedures for optimising SVM hyperparameters

Tuning the regularisation and kernel hyperparameters is a vital step in optimising the generalisation performance of kernel methods, such as the support vector machine (SVM). This is most often performed by minimising a resampling/cross-validation based model selection criterion, however there seems little practical guidance on the most suitable form of resampling. This paper presents the results of an extensive empirical evaluation of resampling procedures for SVM hyperparameter selection, designed to address this gap in the machine learning literature. Wetested 15 different resampling procedures on 121 binary classification data sets in order to select the best SVM hyperparameters. Weused three very different statistical procedures to analyse the results: the standard multi-classifier/multidata set procedure proposed by Demˇsar, the confidence intervals on the excess loss of each procedure in relation to 5-fold cross validation, and the Bayes factor analysis proposed by Barber. We conclude that a 2-fold procedure is appropriate to select the hyperparameters of an SVM for data sets for 1000or more datapoints, while a 3-fold procedure is appropriate for smaller data sets

Peer-selected "best papers" - are they really that "good"?

Background Peer evaluation is the cornerstone of science evaluation. In this paper, we analyze whether or not a form of peer evaluation, the pre-publication selection of the best papers in Computer Science (CS) conferences, is better than random, when considering future citations received by the papers. Methods Considering 12 conferences (for several years), we collected the citation counts from Scopus for both the best papers and the non-best papers. For a different set of 17 conferences, we collected the data from Google Scholar. For each data set, we computed the proportion of cases whereby the best paper has more citations. We also compare this proportion for years before 2010 and after to evaluate if there is a propaganda effect. Finally, we count the proportion of best papers that are in the top 10% and 20% most cited for each conference instance. Results The probability that a best paper will receive more citations than a non best paper is 0.72 (95% CI = 0.66, 0.77) for the Scopus data, and 0.78 (95% CI = 0.74, 0.81) for the Scholar data. There are no significant changes in the probabilities for different years. Also, 51% of the best papers are among the top 10% most cited papers in each conference/year, and 64% of them are among the top 20% most cited. Discussion There is strong evidence that the selection of best papers in Computer Science conferences is better than a random selection, and that a significant number of the best papers are among the top cited papers in the conference.Peer evaluation is the cornerstone of science evaluation. In this paper, we analyze whether or not a form of peer evaluation, the pre-publication selection of the best papers in Computer Science (CS) conferences, is better than random, when considering fu103112sem informaçãosem informaçã

Processos de Decisão de Markov: um tutorial

Há situações em que decisões devem ser tomadas em seqüência, e o resultado de cada decisão não é claro para o tomador de decisões. Estas situações podem ser formuladas matematicamente como processos de decisão de Markov, e dadas as probabilidades dos valores resultantes das decisões, é possível determinar uma política que maximize o valor esperado da seqüência de decisões. Este tutorial descreve os processos de decisão de Markov (tanto o caso completamente observável como o parcialmente observável) e discute brevemente alguns métodos para a sua solução. Processos semi-Markovianos não são discutidos

Inclusion policies in higher education: evaluation of student performance based on the Enade from 2012 to 2014

Este artigo compara as notas obtidas nos Enades de 2012 a 2014 por alunos que entraram no ensino superior via cotas, receberam bolsa ProUni ou empréstimo via Fies com a nota de seus colegas de classe que não receberam o benefício correspondente. A comparação é feita usando a diferença entre as médias das notas padronizadas dos exames gerais e específicos do Enade. O artigo define um limite de equivalência como sendo a diferença da média das notas dos 5% dos alunos com resultados logo acima da média e dos 5% dos alunos logo abaixo da média de todos os alunos. Diferenças abaixo desse valor foram consideradas sem importância prática. Tendo em vista essa definição, alunos cotistas tiveram desempenho equivalente ao de seus colegas de classe não cotistas, assim como os alunos que recebem empréstimo do Fies. Alunos que recebem bolsa do ProUni tiveram desempenho superior ao dos seus colegas de classe.This paper compares the grades on the Enade exams from 2012 to 2014 of students who were accepted into higher education through affirmative action, who received the ProUni scholarship, or who received the government sponsored student loans (FIES) with the grades of students in the same class who did not receive the corresponding benefit. The mean of the standardized exam grades (both general and specific exams) are used for the comparison. The paper defines a limit of equivalence between grades as the difference of the mean grade of the 5% students with grades just above the mean and of the 5% students with grades just below the mean of all students. Differences below this limit of equivalence are considered of no practical relevance. Under this definition, the performance of students who were accepted through affirmative action is equivalent to that of their classmates who did not receive such benefit, as is the performance of students who received the FIES student loan. Students who received the ProUni scholarship performed better than their classmates